Migrate gym to gymnasium by montplaisir · Pull Request #6309 · Unity-Technologies/ml-agents

montplaisir · 2026-07-03T18:17:46Z

Proposed change(s)

Merge these PRs related to gymnasium:

Migrate from Gym to Gymnasium (gymnasium>=0.26.0)
Upgrade Pettingzoo to 1.24.0
Relax the dependencies for Numpy

Co-authored-by: @alexander-zap
Co-authored-by: @vadim0x60
Co-authored-by: @AmineAndam04
Co-authored-by: @Zach-Attach

Types of change(s)

Checklist

Added tests that prove my fix is effective or that my feature works
Updated the changelog (if applicable)
Updated the documentation (if applicable)
Updated the migration guide (if applicable)

u-pr

💡 Harness Review

The API migration is mostly coherent, but a few integration gaps remain: the new PettingZoo reset(seed=...) path does not actually seed anything, and some shipped examples/docs still target contracts this PR removes.

The PettingZoo migration is incomplete across the published examples/docs. com.unity.ml-agents/Documentation~/Python-PettingZoo-API.md and ml-agents-envs/colabs/Colab_PettingZoo.ipynb still unpack env.last() as (observation, reward, done, info), while this PR changes the wrapper to return (observation, reward, terminated, truncated, info). The generated API doc also still advertises the old reset()/seed()/render(mode=...) surface. After this merges, users following the repo’s own guides will hit unpacking errors and try methods/signatures that no longer exist. Please update those docs/examples in the same change.

_{Reviewed commit e4ba566f}

_{🤖 Helpful? 👍/👎}

OliviaBayley

Added a few suggestions for style, but otherwise looks good to me 👍

OliviaBayley · 2026-07-06T08:38:11Z

 - **External Communicator** - which connects the Learning Environment with the Python Low-Level API. It lives within the Learning Environment.
 - **Python Trainers** which contains all the machine learning algorithms that enable training agents. The algorithms are implemented in Python and are part of their own `mlagents` Python package. The package exposes a single command-line utility `mlagents-learn` that supports all the training methods and options outlined in this document. The Python Trainers interface solely with the Python Low-Level API.
- **Gym Wrapper** (not pictured). A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by OpenAI called [gym](https://github.com/openai/gym). We provide a gym wrapper in the `ml-agents-envs` package and [instructions](Python-Gym-API.md) for using it with existing machine learning algorithms which utilize gym.
+- **Gym Wrapper** (not pictured). A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called [gymnasium](https://gymnasium.farama.org/) (formerly OpenAI `gym`). We provide a gym wrapper in the `ml-agents-envs` package and [instructions](Python-Gym-API.md) for using it with existing machine learning algorithms which utilize gymnasium.


Suggested change

- **Gym Wrapper** (not pictured). A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called [gymnasium](https://gymnasium.farama.org/) (formerly OpenAI `gym`). We provide a gym wrapper in the `ml-agents-envs` package and [instructions](Python-Gym-API.md) for using it with existing machine learning algorithms which utilize gymnasium.

- **Gym Wrapper** (not pictured). A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called [gymnasium](https://gymnasium.farama.org/) (formerly OpenAI `gym`). Unity provides a gym wrapper in the `ml-agents-envs` package and [instructions](Python-Gym-API.md) for using it with existing machine learning algorithms which utilize gymnasium.

OliviaBayley · 2026-07-06T08:40:27Z

 # Unity ML-Agents Gym Wrapper

-A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by OpenAI called `gym`. For more information on the gym interface, see [here](https://github.com/openai/gym).
+A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called `gymnasium` (formerly known as OpenAI `gym`). For more information on the gymnasium interface, see the [Gymnasium Documentation](https://gymnasium.farama.org/index.html) and [Gymnasium Github repository](https://github.com/Farama-Foundation/Gymnasium).


Suggested change

A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called `gymnasium` (formerly known as OpenAI `gym`). For more information on the gymnasium interface, see the [Gymnasium Documentation](https://gymnasium.farama.org/index.html) and [Gymnasium Github repository](https://github.com/Farama-Foundation/Gymnasium).

A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called `gymnasium` (formerly known as OpenAI `gym`). For more information on the gymnasium interface, visit the [Gymnasium Documentation](https://gymnasium.farama.org/index.html), and the [Gymnasium Github repository](https://github.com/Farama-Foundation/Gymnasium).

OliviaBayley · 2026-07-06T08:41:10Z

+A common way in which machine learning researchers interact with simulation environments is via a wrapper provided by the Farama Foundation called `gymnasium` (formerly known as OpenAI `gym`). For more information on the gymnasium interface, see the [Gymnasium Documentation](https://gymnasium.farama.org/index.html) and [Gymnasium Github repository](https://github.com/Farama-Foundation/Gymnasium).

-We provide a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gym. Our wrapper provides interfaces on top of our `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.
+We provide a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gymnasium. Our wrapper provides interfaces on top of our `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.


Suggested change

We provide a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gymnasium. Our wrapper provides interfaces on top of our `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.

The ML Agents package provides a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gymnasium. This wrapper provides interfaces on top of the `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.

OliviaBayley · 2026-07-06T08:42:27Z

+## Training with Stable-Baselines3

-OpenAI provides a set of open-source maintained and tested Reinforcement Learning algorithms called the [Baselines](https://github.com/openai/baselines).
+[Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) (SB3) is a set of reliable, actively maintained implementations of reinforcement learning algorithms in PyTorch. It is the community successor to OpenAI Baselines and, like this wrapper, is built on the Farama Foundation `gymnasium` API, so ML-Agents environments can be trained with it directly.


Suggested change

[Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) (SB3) is a set of reliable, actively maintained implementations of reinforcement learning algorithms in PyTorch. It is the community successor to OpenAI Baselines and, like this wrapper, is built on the Farama Foundation `gymnasium` API, so ML-Agents environments can be trained with it directly.

[Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) (SB3) is a set of reliable, actively maintained implementations of reinforcement learning algorithms in PyTorch. It is the community successor to OpenAI Baselines and, like this wrapper, is built on the Farama Foundation `gymnasium` API, so you can train ML-Agents environments with it directly.

OliviaBayley · 2026-07-06T08:43:11Z

+### Example - PPO

-Next, create a file called `train_unity.py`. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, see [here](Learning-Environment-Executable.md). Note that because of limitations of the DQN baseline, the environment must have a single visual observation, a single discrete action and a single Agent in the scene. Add the following code to the `train_unity.py` file:
+To train an agent with PPO on a single-agent environment, create a file called `train_unity.py` with the following code. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, see [here](Learning-Environment-Executable.md).


Suggested change

To train an agent with PPO on a single-agent environment, create a file called `train_unity.py` with the following code. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, see [here](Learning-Environment-Executable.md).

To train an agent with PPO on a single-agent environment, create a file called `train_unity.py` with the following code. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, refer to [Using an Environment Executable](Learning-Environment-Executable.md).

Matching link text to destination title

OliviaBayley · 2026-07-06T08:44:29Z

-### Other Algorithms
-
-Other algorithms in the Baselines repository can be run using scripts similar to the examples from the baselines package. In most cases, the primary changes needed to use a Unity environment are to import `UnityToGymWrapper`, and to replace the environment creation code, typically `gym.make()`, with a call to `UnityToGymWrapper(unity_environment)` passing the environment as input.
+Use the `"MlpPolicy"` for environments with vector observations and the `"CnnPolicy"` for environments with visual (image) observations.


Suggested change

Use the `"MlpPolicy"` for environments with vector observations and the `"CnnPolicy"` for environments with visual (image) observations.

Use the `MlpPolicy` for environments with vector observations, and the `CnnPolicy` for environments with visual (image) observations.

We should avoid quotation marks in the docs where possible

OliviaBayley · 2026-07-06T08:44:48Z

-  MPI = None
+### Training on multiple environments in parallel

+SB3 can train on several environment instances at once using a vectorized environment. Each Unity instance must use a distinct `base_port` so the instances do not conflict:


Suggested change

SB3 can train on several environment instances at once using a vectorized environment. Each Unity instance must use a distinct `base_port` so the instances do not conflict:

SB3 can train on several environment instances at once using a vectorized environment. Each Unity instance must use a distinct `base_port` so the instances don't conflict:

OliviaBayley · 2026-07-06T08:46:06Z

 First, open `dopamine/atari/run_experiment.py`. Alternatively, copy the entire `atari` folder, and name it something else (e.g. `unity`). If you choose the  copy approach, be sure to change the package names in the import statements in `train.py` to your new directory.

-Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Baselines example. At the top of the file, insert
+Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Stable-Baselines3 examples above. At the top of the file, insert


Suggested change

Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Stable-Baselines3 examples above. At the top of the file, insert

Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Stable-Baselines3 examples in the previous section. At the top of the file, insert

OliviaBayley · 2026-07-06T08:47:23Z


 ## Run Google Dopamine Algorithms

+> **Note:** The walkthrough below was written for an older, OpenAI `gym`-based release of Dopamine. This wrapper now follows the Farama Foundation `gymnasium` API: `reset()` returns `(observation, info)` and `step()` returns `(observation, reward, terminated, truncated, info)`. Recent versions of Dopamine target `gymnasium` and are compatible with the wrapper, but the exact file names, module paths, and configuration steps described here may differ from the version you install. Treat this section as a general guide rather than a step-by-step recipe.


Suggested change

> **Note:** The walkthrough below was written for an older, OpenAI `gym`-based release of Dopamine. This wrapper now follows the Farama Foundation `gymnasium` API: `reset()` returns `(observation, info)` and `step()` returns `(observation, reward, terminated, truncated, info)`. Recent versions of Dopamine target `gymnasium` and are compatible with the wrapper, but the exact file names, module paths, and configuration steps described here may differ from the version you install. Treat this section as a general guide rather than a step-by-step recipe.

> [!NOTE] The following walkthrough was written for an older, OpenAI `gym`-based release of Dopamine. This wrapper now follows the Farama Foundation `gymnasium` API: `reset()` returns `(observation, info)` and `step()` returns `(observation, reward, terminated, truncated, info)`. Recent versions of Dopamine target `gymnasium` and are compatible with the wrapper, but the exact file names, module paths, and configuration steps described here may differ from the version you install. Treat this section as a general guide rather than a step-by-step recipe.

Adding formatting for callout box

Update gym to gymnasium

e4ba566

u-pr Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread ml-agents-envs/mlagents_envs/envs/unity_pettingzoo_base_env.py

Comment thread ml-agents-envs/setup.py

montplaisir added 2 commits July 3, 2026 14:51

Address comments by U-PR.

6a0a441

fix pre-commit flake8 fail

b1ea662

montplaisir mentioned this pull request Jul 3, 2026

Multi-agent gym prototype #5481

Closed

update changelog

75e62d9

montplaisir requested a review from OliviaBayley July 3, 2026 20:22

montplaisir changed the title ~~Update gym to gymnasium~~ Migrate gym to gymnasium Jul 3, 2026

OliviaBayley approved these changes Jul 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate gym to gymnasium#6309

Migrate gym to gymnasium#6309
montplaisir wants to merge 4 commits into
developfrom
update-gymnasium

montplaisir commented Jul 3, 2026 •

edited

Loading

Uh oh!

u-pr Bot left a comment

Uh oh!

Uh oh!

Uh oh!

OliviaBayley left a comment

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

OliviaBayley Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	We provide a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gymnasium. Our wrapper provides interfaces on top of our `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.
	The ML Agents package provides a gym wrapper and instructions for using it with existing machine learning algorithms which utilize gymnasium. This wrapper provides interfaces on top of the `UnityEnvironment` class, which is the default way of interfacing with a Unity environment via Python.

	[Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) (SB3) is a set of reliable, actively maintained implementations of reinforcement learning algorithms in PyTorch. It is the community successor to OpenAI Baselines and, like this wrapper, is built on the Farama Foundation `gymnasium` API, so ML-Agents environments can be trained with it directly.
	[Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) (SB3) is a set of reliable, actively maintained implementations of reinforcement learning algorithms in PyTorch. It is the community successor to OpenAI Baselines and, like this wrapper, is built on the Farama Foundation `gymnasium` API, so you can train ML-Agents environments with it directly.

	To train an agent with PPO on a single-agent environment, create a file called `train_unity.py` with the following code. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, see [here](Learning-Environment-Executable.md).
	To train an agent with PPO on a single-agent environment, create a file called `train_unity.py` with the following code. Then create an `/envs/` directory and build the environment to that directory. For more information on building Unity environments, refer to [Using an Environment Executable](Learning-Environment-Executable.md).

	Use the `"MlpPolicy"` for environments with vector observations and the `"CnnPolicy"` for environments with visual (image) observations.
	Use the `MlpPolicy` for environments with vector observations, and the `CnnPolicy` for environments with visual (image) observations.

	SB3 can train on several environment instances at once using a vectorized environment. Each Unity instance must use a distinct `base_port` so the instances do not conflict:
	SB3 can train on several environment instances at once using a vectorized environment. Each Unity instance must use a distinct `base_port` so the instances don't conflict:

	Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Stable-Baselines3 examples above. At the top of the file, insert
	Within `run_experiment.py`, we will need to make changes to which environment is instantiated, just as in the Stable-Baselines3 examples in the previous section. At the top of the file, insert


		## Run Google Dopamine Algorithms

		> Note: The walkthrough below was written for an older, OpenAI `gym`-based release of Dopamine. This wrapper now follows the Farama Foundation `gymnasium` API: `reset()` returns `(observation, info)` and `step()` returns `(observation, reward, terminated, truncated, info)`. Recent versions of Dopamine target `gymnasium` and are compatible with the wrapper, but the exact file names, module paths, and configuration steps described here may differ from the version you install. Treat this section as a general guide rather than a step-by-step recipe.

Uh oh!

Conversation

montplaisir commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed change(s)

Types of change(s)

Checklist

Uh oh!

u-pr Bot left a comment

Choose a reason for hiding this comment

💡 Harness Review

Uh oh!

Uh oh!

Uh oh!

OliviaBayley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

montplaisir commented Jul 3, 2026 •

edited

Loading